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Performance Funding: Tennessee's Noble Experiment In Promoting 
Program Quality Through Assessment of Outcomes 



Tennessee is the first state to employ a performance 
criterion in funding higher education. Information on program 
quality and improvement efforts derived from comprehensive 
program evaluations is a factor in determining each state 
institution's budgetary allocation. Kellogg funding has 
assisted University of Tennessee, Knoxville faculty and 
administrators to assess C\e impact of the performance funding 
initiative and suggest directions for further refinement of 
the performance standards. Despite legitimate criticisms, the 
initiative has strengthened the academic program review process 
and stimulated increased use of outcome information in 
departmental decision-making as well as campus-wide strategic ' 
planning and resource allocation. 
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PERFORMANCE FUNDING: TENNESSEE'S NC3LE EXPERIMENT IN 
PROMOTING PROGRAM QUALITY THROUGH ASSESSMENT OF OUTCOMES 



Assessing Program Quality in Higher Education 

Institutions of higher education have been relatively free to chart 
their own courses and manage their own affairs since the Dartmouth College 
decision in 1819 (Marcus et al., 1983). The ivied walls of academe were not 
penetrated to a significant degree by the legions of evaluators that marched 
into public schools and other social agencies in the Sixties and Seventies 
to assess the effectiveness of programs supported with public funds. But 
with the Fdghties have come reduced confidence in the entire system of pub- 
licly assisted education in America, and a critical need to make well-in- 
formed decisions concerning the appropriate allocation of scarce resources 
among a variety of social services that purport to enhance the quality of 
life. These forces have counteracted any immunity from public scrutiny that 
colleges and universities may have enjoyed in the past, and today there is in- 
creasing recognition of the need for comprehensive program evaluation in insti- 
tutions of higher education. 

Evaluation in higher education has traditionally been formative rather 
than summative, concentrating on what George Kuh (1981) has called context 
variables — proportion of faculty with the doctorate, quality of students as 
defined by scores on. entrance exams, size of the library collection, and expen- 
diture per student rather than on program outcomes such as student achieve- 
ment in general education and the major field. But with falling enrollments 
and steady state financing has come the realization that institutions of higher 
education cannot be all things to all people; rather each must clearly articu- 
late its own mission, determine those things it can do best, and focus resour- 
ces on enhancing strengths in those areas. Comprehensive program evaluation — 
comprehensive in that programs are monitored from the setting of objectives 
through the allocation -^nd utilization of resources for implementation, to 
the measurement of outcomes related to the objectives — can provide the evi- 
dence of quality that is essential in making strategic decisions concerning 
the identity of programs to emphasize, combine with others, or eliminate. 

Enrollment-based formula funding for public higher education has never 
succeeded in providing concomitant increases in program quality as enrollments 
increased. However, the fact that the current interest in improving public 
education comes at a time when both enrollment and price increases have reached 
a plateau may mean that an unusual opportunity exists for addressing qualita- 
tive concerns. Throughout the country there is increasing interest in alter- 
natives to enrollment-based funding that reward instituiiions for assessing and 
improving program quality. 

In 1979 the Tennessee Higher Education Commission (THEC) undertook what 
some observers regard as a noble experiment. Under the leadership of Grady 
Bogue and Wayne Brown (1982) a performance funding policy was instituted that 
applies to all public colleges and universities: Up to 5 percent of an insti- 
tution's annual state allocation for instruction is awarded on the basis of 
its ability to demonstrate accomplishments in five performance areas. 
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According to the current Instructional Evaluation Schedule (TUEC, 1983), pro- 
gram quality is demonstrated by: 

1) calculating the percentage of programs eligible for accreditation 
that are accredited. 

2) calculating the percentage of programs, which, within a five year 
period, have undergone peer review and/or have administered to majors 

a comprehensive fiel<;i exam. Maximum credit for this standard is awarded 
if student performance on the field exam improves over time or exceeds 
the performance of students in similar programs at comparable institu- 
tions. 

3) measuring value added via the general education component of the 
curriculum using the ACT COMP exam, and demonstrating that the per- 
formance of seniors exceeds the mean of value added computed for seniors 
at a group of comparable institutions.. 

4) conducting surveys of enrolled students, alumni, community members, 
and/or employers and demonstrating that generalizations about the quality 
of academic programs or services derived from the surveys have formed the 
bases for specific improvements in campus programs /services . 

5) implementing a campus-wide plan Tor instructional improvement based 
on information derived from procedures 1-4 above, as well as other 
sources . 



The THEC has assigned the following weights to the five instructional 
evaluation variable^: 1-25 percent, 2-30 percent, 3-25 percent, 4-10 per- 
cent, 5-10 percent. In an application of the performance funding mechanism,' 
an institution receiving a $60 million allocation for instruction from the 
state in 1984 could receive an additional 5 percent, or $3 million, for pre- 
paring a report illustrating full accomplishment of the five performance 
standards. 

Tennessee's noble experiment draws nods of approval from accountability 
advocates. But faculty, to whom the responsibility for program evaluation 
must be assigned, are uneasy about the new emphasis on measuring outcomes. 
Their experience tells them that measuring instruments are costly and tirae- 
consuming to develop, and often lack reliability and validity. 

Defining quality in higher education has never been easy. Ideally each 
institution should consider its o\m mission and develop its own standards 
for evaluating program qtiallty. Since the performance funding program is 
meant to be applied to every one of the diverse institutions in Tennessee's 
system of higher education, from technical institutes and community colleges 
to the state's research institution — The University of Tennessee, Knoxville — 
the standards established were, inevitably, the product of numerous compro- 
mises worked out in a political environment. 

In 1981 administrators at the University of Tennessee, Knoxville (UTK) 
had become sufficiently concerned about the potential impact of performance 
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funding on the institution to undertake a formal investigation of the THEC \. 
program. This effort was furthered by the selection of UTK as one of seven ^\ 
institutions in the country to receive a small grant from the W. K. Kellogg 
Foundation, through the National Center for Higher Education Management Sys- 
tems (NCHEMS), for "increasing the use of student outcome information in pro- 
gram planning and decision-making." 

The NCHEMS-Kellcgg Project at UTK 

The student outcomes UTK chose to study under the auspices of the Kel- 
logg grant were those specified in the THEC Instructional Evaluation Schedule: 
Achievement in general education, achievement in the major field, and opinion 
concerning the quality of academic programs and services/ A separate task 
force composed of associate deans and faculty members with appropriate exper- 
tise was appointed to study each of the specified outcome areas . The three 
groups met at 2- to 3-week intervals over the period January 30 - May 31, 1982. 
Each reviewed measurement methodologies available in the areas of concern and 
made recommendations for applying one or more of these at UTK. 

The task force studying general education concluded that while it does 
not cover all skills and understandings general education at UTK is designed 
to convey, the ACT COMP (College Outcome Measures Project) exam could provide 
useful information for evaluating the effectiveness of the general education 
curriculum. Subsequently a testing program was initiated that involves ad- 
ministration of the CObT exam annually to representative samples of freshmen and 
seniors so that value added by the collegiate experience in general education 
can be estimated. 

The task force appointed to look into the area of opinion measurement ^ 
found that a popular commercial opinionaire had been administered to approp- 
riate samples of enrolled students for two years, but survey findings had not 
been applied to improve campus programs. The task force determined that per- 
ceived usefulness could be increased if faculty and students became involved .j*? 
in designing their own survey instrument. A faculty member in sociology and 
another in political science pooled their expertise in survey research to 
develop a questionnaire with contributions from faculty, students, and admin- 
istrators. The instrument was administered for the first time in May 1983 and 
was completed by 70 percent of a sample of 2200 enrolled students. 

The task force investigating the measurement of student achievement in 
the major field was not satisfied with the reliance of the THEC instructional 
evaluation program on the use of scores on comprehensive exams as the be?t in- 
dicators of achievement in the major. They pointed out that even in those in- 
stances in which professional associations had developed reliable and valid 
instruments for measuring the competence of program graduates, often the 
national system of reporting exam scores did not permit use of the scores In 
program evaluation. That is, reporting a single total score provides no in- 
formation about strengths and weaknesses in specific areas of the discipline. 
Some national associations employ a policy that prohibits the release of sub- 
scores to inp.titutions. 

The task forces recommended that Kellogg funds be used to support pilot 
projects — at least one in each college enrolling undergraduates — that 
would give individual units an opportunity to develop their own methodoloj^y 
for assessing program quality. Deans were invited to submit proposals, and 
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on January 1983 fourteen units in nine colleges received awards of $350 to 
$1500 to support pilot efforts to collect and use student outcoine information. 
Seven units proposed to select or design and administer a comprehensive exam 
in the majpr field; seven units elected to survey enrolled students, alumni, 
or employers to gather evaluative data. 

Members of all three task forces agreed that appropriately designed and 
systematically utilized instruments could provide information about student 
outcomes that would be quite valuable in assessing program quality. Having 
established as a result of their intensive investigations a sense of ownership 
of the instructional evaluation program, task force members wanted to insti- 
tutionalize the most desirable aspects of that program. Consequently, they 
recommended that student outcome information be included in the evaluative 
data gathered for the University's comprehensive program review process. New 
language was prepared for the self-study guidelines that calls for the inclu- 
sion of outcome information such as comprehensive test scores; student opinion 
concerning courses, curriculum, methods of teaching, and services such as ad- 
vising; and data on placement of graduates. This step ensures that every pro- 
gram in the University Vvill give some attention to outcomes at least once 
every five to seven years when the program goes through its intensive review 
by internal and external consultants. 

Finally, the NCHEMS-Kell ogg project task forces recommended that the pro- 
gram reviews with their new emphasis on outcomes be used by the recently formed 
Planning and Budgeting Coordinating Committee to make strategic decisions con- 
cerning programs to be strengthened through infusion of institutional resources, 
programs to be maintained or combined with others, and those to be re.duced in 
scope or size. 

Results of Implementing NCHEMS-Kellogg Project Recommendations 

The stated purpose of the NCHEMS-Kellogg initiative was to increase the 
use of student outcome information in planning and decision-making. NCHEMS 
and Kellogg assumed — and specified in the request for proposals that thl.q would 
be the case — that each of the seven institutions selected for the national 
project had at its disposal quantities of student outcome information that 
with a little reworking and promotion could be utilized more extensively. The 
UTK proposal described examples of such information on campus. However, once 
the project was unden^ay, interviews with deans, department heads, and faculty 
provided evidence that few were satisfied x^ith the quality of the outcome in- 
formation at their disposal. Previous administrations of the COMP exam had 
been to seniors only, and to samples of seniors that were not considered repre- 
sentative of colleges or of the senior class. Scores from most nationally 
standardized exams in major fields were ignored because little information is 
conveyed by total scores alone, or because the exams were not considered con- 
tent valid for assessment of the relevant UTK programs. Data from commercial 
survey instruments administered to students were ignored because items in the 
surveys were not considered appropriate for use in evaluating UTK programs. 

Changes in instruments, in data-gathering methodology, in ways of analy- 
sing responses and interpreting results were needed before outcome information 
could acquire the credibility essential for utilization. Implementation of 
task force recommendations concerning administration of the COMP exam and 
design of local measures of opinion and achievement in order to increase face 
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and content validity produced significant examples of utilization of outcome 
information at the University of Tennessee, Knoxville. 

Impact on Curriculum and Instruction 

The score reports prepared by ACT following the testing of freshmen and 
seniors in 1983 with the COMP exam have gererated substantial interest among 
the academic deans, and among the department: heads and faculty who have seen 
the reports. The measure of value added — score gain from the freshman to 
senior years — provided encouragement for curriculum planners: Mean score 
gain at UTK is well above the mean of score gains at peer institutions. How- 
ever, percentile rankings on the two subscales Functioning in Social Institu- 
tions and Solving Problems are not as high as the faculty would like to see 
them. While there is no sound basis, nor is there an immediate need, for 
making precipitous changes in general education distribution requirements on 
the basis of the COMP exam scores^ there is considerable interest in (1) ana- 
lyzing relationships between COMP scores and such factors as courses taken, 
time on task, and participation in internships, interdisciplinary courses, 
and other special experiences; and (2) reviewing future reports from ACT to 
see if the pattern of subscore strengths and weaknesses obtained in 1983 is 
maintained in subsequent years. A group of faculty with interest in analy- 
zing large data sets has been organized to carry out the analytical studies. 

Preliminary data analyses indicate that participation in student profes- 
sional organizations is associated both with high scores and high score gain 
on the COMP exam. This finding suggests, obviously, that increased emphasis 
on student participation in professional organizations might enhance the edu- 
cational experience of students on campus. The UTK Coordinating Committee on 
General Education, as well as the Board of Deans and others interested in pro- 
gram improvement will continue to consider the COMP results and the unfolding 
picture of correlates of achievement as they contemplate broad changes in cur- 
ricula and course work necessitated by increasing selectivity in admissions 
standards and a planned conversion from an academic calendar based on quar- 
ters to a semester system. 

Study of the COMP exam itself has proven to be a faculty development 
activity that holds much promise. Dr. Aubrey Forrest of ACT has been on cam- 
pus twice under the auspices of the NCHEMS/Kellogg Project, and his presenta- 
tions have been widely acclaimed by participants. The design of the COMP 
exam is quite interesting: Each item has been constructed to assegs skills 
in a process area (communicating, solving problems, clarifying values) as 
well as a content area (social science, science/technology, the arts). More- 
over, the test items require the student to apply higher order intellectual 
skills such as analysis, synthesis, and evaluation, not just recognition and 
recall. Thus careful study of the exam and the rationale on which it is 
based suggests to faculty ways of teaching and testing students that foster 
development of the more complex skills. More faculty development of this 
kind is contemplated. 

\*Jhile changes in general education will be made slovjly due to the 
institution-wide implications of many such endeavors, changes in curricula, 
instruction, and supporting services associated with individual programs can 
be made more quickly and easily. Administration of the "Student Satisfac- 
tion Survey" by Professors Kent Van Liere and William Lyons in Spring 1983 
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revealed some student assessments of the quality of programs and services 
that proved disquieting to administrators and faculty in several units. Stu- 
dents in the College of Communications provided ratings of "availability of 
advisor" and "availability of required courses in the major" that were below 
University means for those items. The Communications dean was sufficiently 
concerned about these findings to take immediate steps to improve advising 
and make adjustments to give more students access to required courses. A 
full-time advisor for freshmen and sophomores was added to the dean's staff, 
and the number of advisees assigned to each faculty member was reduced. 

Student Satisfaction Survey results in the colleges of Business and 
Engineering, and the Department of Political Science prompted faculty in those 
units to initiate their own follow-up student surveys. The very detailed 
Engineering survey was administered by mail in February 1984, so results have 
not yet been compiled. In the business school a faculty member with exper- 
ience in using a TELLUS machine that permits responses to ten items to be 
entered electronically and provides immediate tabulation of results, admin- 
istered questions to over 400 students during preregistration. That endeavor - 
furnished evidence of some dissatisfaction with student advising, and the 
appropriate associate de:in undertook a thorough review of advising procedures 
and practices in the College. Advisors and all faculty have been sensitized 
to the needs for increased interaction with advisors and faculty that were 
indicated by students' responses to both the initial survey and the college 
follow-up. 

The department head and faculty in the Department of Political Science 
were concerned because the Spring 1983 administration of the "Student Satis- 
faction Survey" indicated that students were somewhat dissatisfied with the 
quality of instruction provided by graduate teaching assistants. The depart- 
mental committee on undergraduate instruction designed a follow-up study that 
involved administering to all undergraduate students taking political science 
classes in Fall 1983 the portion of the "Student Satisfaction Survey" dealing . 
with the classroom experience. Following analysis of 1100 student responses 
the department head had, for the first time, comparative data on the perceived 
teaching effectiveness of every faculty member and every graduate teaching 
assistant (GTA) . He shared the comparative data individually with each faculty 
member, and he believes this process will encourage faculty to take specific 
steps to improve their teaching. Indeed GTAs as a group did receive lower 
effectiveness ratings than faculty. Ways of responding to this finding are 
being explored — larger lecture sections taught by outstanding faculty with 
discussion sections staffed by GTAs. Specific changes in content and methods 
of instruction are being planned for the introductory course in political 
science. 

The pilot project — a survey — carried out in the Department of Tex- 
tiles, Merchandising, and Design pointed to a need, once again, for improve- 
ment in student advising. In response a new curriculum planning sheet for 
majors was designed. 

In several units the results derived from pilot projects confirmed pre- 
sent practice, e.g., high scores on the Graduate Record Advanced Tests by 
seniors in the departments of psychology, history, botany, and microbiology 
encouraged those faculties to continue current patterns of course work and 
classroom instruction. But in the departments of Geography and Food Tech- 
nology and Science, the faculty decision to design their o\m comprehensive 
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exam for seniors had profound effects. In order to determine the content of 
their exams the faculties had to consider in a more intensive way than hercjto- 
fore such curriculum matters as the relative emphases given to specialty areas 
of the discipline, the way course sequences fit together and build on each 
other, and the competencies students should possess upon completion of the 
curriculum for program majors. 

The exam developed over the better part of a year by the faculty of the 
Department of Geography has four sections: Physical Geography, Economic 
Geography, Cultural Geography, and Technique. Seniors who took the exam when 
it was administered for the first time in February 1984 attained high scores 
on the Cultural Geography section; their lowest scores were in economic geog- 
raphy. These specific findings and the global picture of scores have caused 
the faculty to undertake several curriculum changes: A course in Gcononic 
geography will be added, cultural geography will be refocused (its emphasis 
in the curriculum will be slightly reduced), and students will be encouraged, 
perhaps required, to take a stronger common core of courses. 

As in Geography, all faculty in the Department of Food Technology and 
Science (a unit in the College of Agriculture) were involved in developing 
their comprehensive exam for seniors. While Geography faculty chose not to 
involve in the design phase UTK faculty with expertise in measurement, the 
Food Tech faculty did schedule several sessions with a Kellogg Project con- 
sultant identified for this purpose. Professor Schuyler Huck from the Depart- 
ment of Educational Psychology and Counseling. Dr. Huck provided general 
guidance in developing multiple choice test items, and assisted in determin- 
ing instrument reliability. In addition to the opportunity for internal re- 
view of instruments by a specialist in measurement, both the departments were 
encouraged to retain two external consultants to review their tests. The two 
faculties identified respected scholars in the discipline and asked those in- 
dividuals to review drafts of their exams and even to give the tests to their 
own students for purposes of obtaining comparative data. 

Analysis of students* test scores also brought about some immediate 
changes in the Food Tech Department. Faculty were not satisfied with the 
scores achieved on the Microbiology and Food Chemistry sections of the test. 
Students were not able to apply their knowledge to solve problems to the ex- 
tent faculty had hoped they would be able to do. Following a series of meet- 
ings, the faculty teaching microbiology and food chemistry courses agreed to 
place much more emphasis on applications in their teaching and in their cJ.ass- 
room tests. 

The Food Tech faculty already have begun to consider v/ays to improve 
their exam for seniors. They want to add a performance measure that would 
take students into the laboratory to solve certain problems. Students' lab 
techniques would be judged as well as their approach to a problem and their 
resolution of it. 

The use of nationally standardized exams to test student achievement for 
purposes of program evaluation can be valuable. At UTK the dean of the College 
of Business asked seniors to take the Business Assessment Test offered by Edu- 
cational Testing Service as part of its Underj;raduate Assessment Program. 
Students' scores gave evidence of particular strength in economics and weak- 
ness in business law. The college faculty had other indications of these 
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anomalies In the curriculum nv^ acted to reduce the three-course requirement 
in economics to two courses t^v aat a requirement in ^business law could be 
added. Notwithstanding this i^ind of example, the experience on this campus 
to date indicates that faculty have a greater intellectual stake in the out- 
comes of testing, and are more likely to undertake improvement initiatives 
based on those outcomes, if they have invested the tim^ to become involved 
in designing their own comprehensive exam. ^ 

On technical grounds one can argue against the use of locally developed 
tests; there are no norms against which to judge local performance, reliabil- 
ities may be questionable, content and predictive validity are difficult to 
demonstrate. But when a test is being used to assess and improve program 
quality , students' scores are aggregated and faculty consider the \implications 
of scores for changing the program rather than for making judgments about the 
relative competence of individual students . Thus the importance of technical 
flaws in the instrument is minimized, and the potentiality for effecting mean-.^, 
ingful program improvement is great. 

Impact on Student Services 

Responses to the Student Satisfaction Survey items on registration and 
availability of courses indicated some dissatisfaction with the registration 
proxiess. The Vice Chancellor for Student Affairs, Howard Aldmon, used this 
finding to support a decision to establish an earlier cut-off date for admis- 
sion to the University so that student demand for classes could be assessed' 
in a more timely fashion and extra sections scheduled to accommodate demand. 
In a further attempt to improve the quality of student advising. Vice Chancel- 
lor Aldmon asked that each dean provide a representative to meet with studenLs 
encountering special problems during the drop/add process. 

Student retention is a matter of increasing concern to campus administra- 
tors across the country. Analysis of Student Satisfaction Survey responses 
for students returning to the University and those not returning two quarters 
later has provided preliminary evidence that satisfaction with the University 
is an important factor in students' decisions to pursue their studies at UTK. 
Higher grade point averages and fewer hours of employment (less than 30 hours 
per week) also are linked with persistence on this campus. Developers of the 
Student Satisfaction Survey found that patterns of student satisfaction varied 
by college. As a consequence, they have provided a profile for each of the 
nine colleges enrolling undergraduates of factors characterizing student aca- 
demic and social satisfaction in that unit. For some colleges social satis- 
faction — interaction with peers (perhaps in student professional organiza- 
tions) or membership in a social fraternity — contributes most to overall 
satisfaction with the University. For the professional schools — nursing, 
business, architecture, engineering — interaction with faculty is a potent 
factor in determining satisfaction. These profiles are being used by college 
faculties to make adjustments they hope will help to increase student reten- 
tion. 



Impact on Institutional Planning and Evaluation 

The University of Tennessee, Knoxville has a carefully crafted ten-year- 
old program review process that is conscientiously implemented by the Provost 
and his staff. Central administrators, including the Chancellor himself, 
ireview the self-study, and reports prepared by internal and external reviewers 
following a 2i$-ciay intensive site visit. Objectives and aspirations of the 
department or program under review and the recommendations of the reviewers 
are considered in University-wide as well as unit planning and in internal re- 
source allocation. 

Prior to 1983 the UTK program review guidelines, like those at many other 
institutions, focused primarily on program input factors as evaluative criteria. 
Qualifications of faculty, quality of students as measured by aptitude tests or 
grade-point averages, and adequacy of facilities and the library collection, 
were included by program faculty in the self-study, and examined by reviewers. 
In July 1983 the director of the UTK Kellogg Project was given the opportunity to 
submit new material for the program review gu^.delines that would provide a 
focus on student outcomes such as placement of graduates, opinion concerning 
quality of the program and supporting services, and achievement in general 
education and the major field. The revised guidelines were put into effect 
in September 1983. -> 

To emphasize the importance of using student outcome information in pro- 
gram evaluation, in January 1984 Provost George \Jheeler asked every academic 
unit to develop a plan for assessing student achievement in the major field 
at least once in the next five y^ars. 

The UTK Planning and Budgeting Coordinating Committee, appointed by Chan- 
cellor Jack Reese in September 1982, has reviewed the Van Liere-Lyons reten- 
tion study in preparation for planning strategies to increase student persis- 
tence at UTK. Information on freshman and senior COMP exam scores accompanied 
by preliminary analyses of correlates of achievement in general education — 
including reasons ^Iven by freshmen for choosing to atterd UTK — has been 
considered by the committee as it seeks to clarify the mission statement of 
the University and communicate it to the clientele the institution hopes to 
serve. During Spring Quarter 1984 the committee will work with Professors Van 
Liere and Lyons and the Learning Research Center staff to develop and admin- 
ister a survey of alumni to obtain their opinions concerning program quality. 
The committee has begun to use the academic program review documents, with ^ 
their new emphasis on student outcome information, in determining which pro- 
grams to strengthen through allocation of additional resources, and which to 
combine with others or terminate. 

In the most tangible and immediately effective illustrations of outcomes 
utilization to date, a reference to the use of student outcome information as 
evidence of program quality was included In the 1984 instructions for use by 
program heads in preparing their annual budget requests. In addition, the 
Planning and Budgeting Coordinating Committee used student outcomes as one of 
the criteria in selecting the campus proposals to be entered in Tennessee 
Governor Lamar Alexander's 1984 state-wide Centers of Excellence competition. 
Previously the criteria for assessing quality in procedures such as these had 
included only input and process variables. 
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An Assessment of Performance Funding 

The task of convincing administrators and faculty of the value of syste- 
matically collecting evidence of program quality and putting it to use re- 
quires strong leadership, and time. 

Powerful objections to the collection and use of outcome information can 
be raised: The process costs too much, faculty have difficulty agreeing on 
program objectives, appropriate instruments are not available, and instruments 
that are available lack acceptable reliability. Representative study and 
planning groups must be established to consider these objections, to review 
measurement methodologies, and to assess the potential benefits of employing 
these strategies. The methodologies must be adapted to fit institutional and 
program missions. Individuals participating in the study groups mast communi- 
cate their findings to the students, faculty, and administrators they repre- 
sent. Ultimately many will recognize that if faculty believe an instrument 
can give them information about strengths and weaknesses of their program, 
some of the technical imperfections in the instrument may be overlooked since 
data will be aggregated for assessment of the program rather than the relative 
competencies of individual students . Conducting pilot tests of alternative 
methodologies will provide an experience base upon which widespread applica- 
tion can be built. 

At UTK the NCHEMS-Kellogg grant provided access to the leadership, the 
time, and some financial assistance for (1) carrying out the studies essen- 
tial to establishing the institution's ownership of its student outcomes 
assessment program, (2) establishing the communication networks necessary to 
inform faculty and .students of the benefits of such a program, and (3) pilot 
testing alternative methodologies for collecting and using outcome information. 
The use of student outcome information in program assessment and improvement 
has been woven into the fabric of institutional planning and decision-making 
at UTK. As a result: of the adoption of appropriate new policies and proce- 
dures, program heads in all colleges are using outcome data to inform the 
process of improving their programs, while central administrators are consider- 
ing these data as they make strategic decisions about program mix and alloca- 
tion of internal resources. 

The analysis of the THEC Instructional Evaluation Schedule which took 
place at UTK with the assistance of the Kellogg grant provided an assessment 
of the impact of performance funding that was generally positive. An ir.sti- 
tution can benefit in many ways from institution-wide application of compre- 
hensive program evaluations that include systematic measurement of student 
outcomes* 

The study of performance funding provided an endorsement of the evalua- 
tive criteria employed in the THEC Schedule. However, serious questions were 
raised about some of the standards of accomplishment associated with the cri- 
teria, and about the relative weights assigned to each. 

Accreditation is a process which certifies that a program has met cer- 
tain minimum standards, but beyond that minimum it does not assist in dif- 
ferentiating programs on the basis of quality. In complex institutions where 
it makes sense for related programs to share competent faculty members rather 
than duplicate the expertise in two or more units, it may be prudent not to 
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pursue accreditation in every program field. For these and other reasons, 
accreditation should be given less weight in performance funding than the THEC 
has given it, and institutions should be given an option to apply for program 
exemptions by submitting detailed explanations, consistent with their, mission, 
for not pursuing accreditation in certain fields* 

Measures of achievement in general education and in the major field are 
appropriate evaluative criteria. However, far too much emphasis is given in 
the current draft of the THEC Instructional Evaluation Schedule to the use of 
scores on comprehensive tests as the principal indicator of achievement. Com- 
prehensive program evaluations, incorporating external peer review, should be 
at the heart of a performance funding program, and each unit should have the 
freedom to select the method (s) to be used in demonstrating that students have 
satisfactorily achieved the learning outcomes established for them by faculty 
in the major field. These methods include testing, obtaining performance rat- 
ings for students and/or program graduates, having external consultants eval- 
uate theses and dissertations, keeping records of placements of graduates in 
related jobs or in accredited graduate programs, and reporting outstanding 
professional accomplishments of graduates. 

Currently the THEC Schedule attaches too little importance to the use of 
opinion surveys and to the overall plan for instructional improvement. Care- 
fully designed surveys, particularly whan carried out in conjunction with 
achievement testing, can provide important clues to the reasons behind high 
or low test scores. In some cases relatively inexpensive changes in student 
advising or field experiences suggested by survey results can substantially 
increase student satisfaction, which is a potent factor in retention — a 
matter of increasing importance to all colleges and universities. 

The nature of the plan for instructional improvement proposed c-^s an 
evaluative criterion by the THEC needs to be defined more precisely. If 
comprehensive program evaluation can be viewed as a central element in per- 
formance funding, each institution's annual plan for instructional improve- 
ment should identify the academic programs reviewed the previous year and 
indicate how recommendations for improvement contained in the reports of 
internal and external reviewers have been addressed by the academic unit, 
its college administration if applicable, and the central administration. 
Actions designed to effect improvements could take many forms, including re- 
allocation of internal resources to provide for additional faculty, or equip- 
ment needs; adjustments i.n curriculum patterns, course structure, or instruc- 
tional methods; provision of faculty development experiences or improved in- 
ternship experiences for students. 

The foregoing discussion of the relative emphases now given to evalua- 
tive criteria by the THEC as opposed to those proposed by investigators at 
UTK may be summarized as follows: 
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Current Proposed 

THEC Standard ' Weight Weight 

Accreditation 25% 10% 

Assessment of Student Achievement 30% 25% 
in the Major Field (to be changed 
to "Comprehensive Program Review") 

Assessment of Student Achievement in 25% 20% 
General Education 

Opinion Surveys 10% 20% 

Plan for Instructional Improvement 10% 25% 

(based on response to program reviews) 



TOTAL 100% 100% 

Institutional resistance to new funding initiatives based on qualita- 
tive assessments can be reduced if ''performance funding" comes from addi- 
tional state allocations for higher education rather than from rearrangements 
of the existing funding base. Assessment and improvement of program and in- 
stitutional quality should be rewarded with new funds in a manner that is 
highly visible. 

This argument can be pursued at the campus level: Deans, department 
heads, and faculty will undertake more willingly the effort needed to col- 
lect outcome information for use in meeting performance funding requirements 
if the institution recognizes and provides tangible rewards for demonstrat- 
ing program quality and attempts to improve it. First tne institution's aca- 
demic leadership can provide personal acknowledgement and public recognition 
of efforts to assess and improve program outcomes. Incentive funds can be 
used to encourage units to undertake instructional improvement activities* 
Most importantly, outcome information can become a highly valued considera- 
tion in planning decisions that result in i:uproved resource allocations. 

Ultimately the highest reward for undertaking the assessment of outcomes 
will be intrinsic: Faculties will obtain evidence that improvements in cur- 
ricula and instruction have fulfilled theix goal of providing the best pos- 
sible education for students. 
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